Testing is a vital component of the software development process, and provides strong support for the production of quality software. Many organizations have not realized the full potential of their testing processes as evidenced by the relative immaturity of these processes. Process improvement models that are currently available, such as the Capability Maturity Model® (CMM), Capability Maturity Model Integration (CMMI), International Organization for Standardization (ISO) 9001, Bootstrap, and SPICE (ISO 15504) have not adequately addressed testing issues. In response, the Testing Maturity ModelSM (TMM) was developed in 1996 at the Illinois Institute of Technology to address testing gaps in these and other known improvements efforts. The purpose of the TMM is to assist software organizations with assessing and improving their software testing processes. The TMM is a compliment to the Capability Maturity Model, and is based on the historical evolution of the testing process and on best industry practices. While a complete understanding of the TMM is beyond the scope of the present discussion, a quick overarea of the TMM is now provided. For more detail on the TMM, please refer to the website http://www.cs.iit.edu/research/tmm.html and to Practical Software Testing: A Process-Oriented Approach by Ilene Burnstein (Springer Press, New York, 1st Edition, 2003).
The TMM has two major components, (1) a TMM hierarchy 100 with a set of five levels 110-150 that define testing capability; and (2) An Assessment Model. Each of the TMM levels 110-150 is composed of Maturity Goals, Maturity Subgoals Activities, and Tasks and Responsibilities. The Assessment Model consists of a Maturity Questionnaire with a web-based component, an Assessment Procedure, and Team Selection and Training Guidelines.
As depicted in FIG. 1 (PRIOR ART), the TMM hierarchy 100 takes a 5-staged approach (like the CMM and the CMMI) to testing maturity for organizations, where the TMM maturity goals make up the five TMM Test Maturity Levels 110-150. The TMM structure hierarchy 100 has Maturity Differentiators between the five levels 110-150, and the test maturity differences help to understand test practices of organizations. The differences in the five stages 110-150 of the TMM hierarchy 100 are now summarized below in TABLE 1.
TABLE 1‘Initial level 1’ vs. ‘Defined level 2’At TMM level 1 110, organizations struggle betweentesting and development/debugging, whereas at TMM level2 120, they recognize and define testing as a distinctdiscipline‘Defined level 2’ vs. ‘Integrated level 3’At TMM level 2 120, organizations start testing withan unpredictable starting situation, whereas at TMMlevel 3 130, they aim to test & plan testing early inthe Software/System development life cycle (SDLC)‘Integrated level 3’ vs. ‘Managed/Measured level 4’At TMM level 3 130, organizations still focus ontesting often & everywhere, whereas at TMM level 4 140,they aim to test less by management/control & qualityrearea‘Managed/Measured level 4’ vs. ‘Optimized level 5’At TMM level 4 140, organizations aim to reduce testeffort by management & control, whereas at TMM level 5150, they aim to reduce effort by automation,statistical testing, but mostly by defect preventionthrough overall SDLC process improvement
It is generally accepted that the TMM concepts are intellectually strong with a sound and comprehensive academic structure. Furthermore, the TMM is aligned with other maturity models such as CMMI, but the TMM improves testing gaps in CMMI and other process models and further provides a basis for evaluating an organization's process testing capabilities (in contrast to the CMMI's focus on evaluating the organization's software creation process.
In practice, an organization may be evaluated using the TMM (or other maturity model) by a skilled individual trained in the maturity model. Specifically, this evaluator reareas the organization according to a series of standards (or requirements) associated with each of the five maturity levels. When an organization satisfies each and every of the requirements for a maturity level, the organization is said to have achieved that particular maturity level. The levels are cumulative in that all of requirements of a lower level must be achieved in order to satisfy the requirements of a higher level. For example, an organization cannot achieve TMM level 3 130 without previously achieving the requirements of TMM level 2 120. In this way, the TMM requires stage-by-stage testing such that an organization needs to show compliance with a lower TMM level before testing can begin to show compliance with a higher level.
The resulting document is generally quite extensive, including detailed analysis of the each of the criteria for each of the stages, along with documentation to verify the ratings. For example, the response for each of the criteria may generally include an explanation for the finding along with supporting documentation for the finding, such as organizational documents showing the existence of various procedures or equipment, as necessitated by the stage requirement.
However, this staged testing methodology limits the practicality of the TMM because it encourages a stage-by-stage compliance, rather than holistic approach to maximize overall testing maturity. Typically, an organization would attempt to satisfy the requirements of one level before proceeding the next higher level even though it is possible to concurrently address the requirements for several levels, even where the requirements from different levels are closely related and could be efficiently addressed together. Other than providing the testing criteria, the TMM also does not help an organization to target specific test improvement opportunities, such as improving testing strategy, test automation, performance testing, requirements management, etc. For these and other reasons, the TMM currently has no large-scale, practical implementation.
For similar reasons, the TMM does not serve as a useful measure of an organization's testing capabilities since it merely provides criteria for completion of a level with no indication of the organization's performance in other levels. For example, if an organization having very mature testing capabilities fails one of the requirements of TMM level 2 120, than that organization would only be TMM level 1 110, even if that organization could otherwise satisfy most requirements for the other TMM levels 2-5 120-150.